Reduce dashboard CPU under sustained metric ingestion - #20736

Description

Long-running AppHosts could consume steadily increasing dashboard CPU while ingesting a fixed metrics workload. Metric retention ranked every retained point in each changed dimension on every insertion, including when the dimension was still below its retention limit. This reproduces independently of exemplars.

Cache each dimension's retained point count and load it from SQLite when an existing dimension is first used. Skip retention cleanup below capacity; when capacity is exceeded, use the existing dimension-order index to delete only the oldest surplus points. Existing cascading foreign keys continue to delete the removed points' exemplars and filtered attributes in the same transaction.

Also:

• Add behavioral coverage for dimension-specific retention, repeated values, database reopening, and recovery after a failed retention transaction.
• Remove SQL-activity assertions and the SQL-only lookup test, retaining assertions on the returned telemetry.
• Add AddHistogramMetricsAtCapacity, preloading 50,000 points per dimension and replaying four exemplars per point. History population and payload construction are outside the measured operation.
• Update the metrics benchmark's schema initialization, async query API calls, and in-process toolchain so it can execute on the current .NET 11 SDK.
• Document the ingestion benchmark.

User-facing usage

Continue starting the AppHost normally, for example with aspire start. No configuration changes are required. The metric retention limit, one-second development export interval, and exemplar behavior are unchanged; steady-state ingestion no longer scans the retained history on every insertion.

Benchmark comparison

Both variants used BenchmarkDotNet 0.15.8, .NET 11 Release builds, Windows x64, the same populated-history workload, three warmup iterations, and ten measured iterations. The baseline used the original retention source from commit 0a42e2c6ae; the fixed variant used this branch's implementation.

The baseline used 20 operations per iteration. The fixed path used 512 operations per iteration to avoid short samples. Results are normalized per export, which adds one point per dimension. The fixed five-dimension result had a standard deviation of 1.073 ms, so the speedup figures should be treated as approximate.

Validation

• Passed: 59 targeted tests, excluding quarantined and outerloop tests:

dotnet test --project tests\Aspire.Dashboard.Tests\Aspire.Dashboard.Tests.csproj --no-launch-profile /p:SkipNativeBuild=true /p:InstallBrowsersForPlaywright=false -- --filter-class "*.SqliteMetricsTests" --filter-class "*.SqliteTelemetryPersistenceTests" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"

• Passed: focused baseline and fixed Release builds. The full benchmark project has pre-existing compile errors in unrelated span/trace benchmark classes that use removed APIs. A temporary MSBuild override excluded those classes for both builds; for the baseline only, it substituted the original metrics-write source without reverting the worktree fix. <scratch> below replaces the local temporary directory.

dotnet build benchmarks\Aspire.Dashboard.Benchmarks\Aspire.Dashboard.Benchmarks.csproj --configuration Release --no-restore /p:SkipNativeBuild=true /p:DirectoryBuildTargetsPath="<scratch>\issue20725-baseline.targets" /p:UseOriginalMetricRetention=true
dotnet build benchmarks\Aspire.Dashboard.Benchmarks\Aspire.Dashboard.Benchmarks.csproj --configuration Release --no-restore --no-incremental /p:SkipNativeBuild=true /p:DirectoryBuildTargetsPath="<scratch>\issue20725-baseline.targets" /p:UseOriginalMetricRetention=false

• Passed: both ingestion benchmark cases with the original and fixed implementations:

dotnet exec artifacts\bin\Aspire.Dashboard.Benchmarks\Release\net11.0\Aspire.Dashboard.Benchmarks.dll --filter '*AddHistogramMetricsAtCapacity*' --strategy Throughput --launchCount 1 --warmupCount 3 --iterationCount 10 --invocationCount 20 --unrollFactor 1 --artifacts '<scratch>\issue20725-benchmark-before' --exporters json
dotnet exec artifacts\bin\Aspire.Dashboard.Benchmarks\Release\net11.0\Aspire.Dashboard.Benchmarks.dll --filter '*AddHistogramMetricsAtCapacity*' --strategy Throughput --launchCount 1 --warmupCount 3 --iterationCount 10 --invocationCount 512 --unrollFactor 1 --artifacts '<scratch>\issue20725-benchmark-after-precise' --exporters json

• Passed: all 12 existing metrics-query benchmark cases as a smoke test:

dotnet exec artifacts\bin\Aspire.Dashboard.Benchmarks\Release\net11.0\Aspire.Dashboard.Benchmarks.dll --filter '*Get*Metrics*' --artifacts '<scratch>\issue20725-benchmark-query-smoke'

• Passed: git diff --check.
• The exact 15-service macOS workload and native AOT publish were not validated.
• No new dependencies or database schema changes.

Fixes #20725

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No
• Did you add public API?
  • Yes
    • If yes, did you have an API Review for it?
      • Yes
      • No
    • Did you add <remarks /> and <code /> elements on your triple slash comments?
      • Yes
      • No
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
    • If yes, have you done a threat model and had a security review?
      • Yes
      • No
  • No